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5 The invention relates to a method and to an apparatus for 
synchronizing data streams containing audio, video and/or 
other data, wherein some of the data streams are recorded in 
a multiplex on a storage medium and other data streams are 
located out of the data stream multiplex. 

10 

Background 

Optical storage media facilitate recording or replaying of 
15 audiovisual (AV) signals that include one or more video and 
audio data streams and/or other data like subtitles and ser- 
vice information. Based on the Digital Versatile Disc (DVD) 
standard published by www.dvdforum.org pre-recorded movie 
titles are replicated for a mass market using read-only op- 
20 tical discs. Following introduction and proliferation of the 
red laser based DVD standard, a blue laser based media sys- 
tem of even higher capacity has recently been published un- 
der the brand name 'Blu-ray Disc 1 denoted BD. 

25 For nowadays DVDs the content providers create videos for 

many different target countries. This encompasses many dif- 
ferent languages and sub-titles. Additional to this lan- 
guage specific versions of AV material there exists the ap- 
plication of multi-angle and multi-story for optical discs. 

30 In BD applications this possibilities will be enlarged by 

introducing different video formats as SDTV and HDTV videos. 
For optical recording and pre-recording all the different 
elementary streams listed above (video-angle, audio- track, 
subtitle etc.) have to be packetized into a single multiplex 

35 transport stream. During playback the decoder is reading 
this multiplex and decodes those packets belonging to the 
streams selected. 
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Invention 

The invention is based on the recognition of the following 
fact. Having an AV multiplex carrying all the available com- 
5 ponents it is not possible to provide additional material, 
like another sound track or another web-page for a later 
time. I.e. when a content provider is using an authoring 
tool for the generation of a BD movie all elementary streams 
have to be known before starting the generation. It is not 
10 possible to add another elementary stream after the AV mul- 
tiplex production. On the other hand, if a further data 
stream shall be added later on, this requires a synchroniza- 
tion with the already existing data streams. 

15 Therefore, a problem to be solved by the invention is to al- 
low synchronizing of data streams recorded in a multiplex on 
a storage medium with further data streams located out of 
this data stream multiplex. 

20 This problem is solved by the method disclosed in claim 1 
and the corresponding apparatus in claim 8 . 

In principle, the inventive method allows synchronizing data 
streams containing video, audio and/ or other data, wherein 
25 some of the data streams are pre-recorded in a multiplex on 
a storage medium, wherein a navigation file comprises de- 
scriptors pointing to parts of said data streams, wherein 
said descriptors define the arrangement in time for said 
data streams by means of data sub stream paths. 

30 

Advantageous additional embodiments of the invention are 
disclosed in the respective dependent claims. 

35 Drawings 



Exemplary embodiments of the invention are described with 
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reference to the accompanying drawing, which shows an exam- 
ple of a play list encompassing different video, audio and 
subtitle stream paths. 



Exemplary embodiments 

Figure 1 shows an example of a play list List_of_PlayI terns 
encompassing different video, audio and subtitle stream 
10 paths with Playltems or SubPlayltems , which have to be de- 
coded for playing back of a stream path. A first stream path 
of Playltems comprises an AV multiplex stream. Furthermore, 
several SubPlayltems describe audio, video and subtitle 
stream paths as depicted. 



15 



20 



As shown, multiple stream paths may exist in parallel on the 
global time axis of the PlayList. A stream path may be de- 
fined leaky by leaving parts of the time axis empty. 

Any stream described by a Playltem or a SubPlayltem may be 
an elementary stream or again a multiplex of streams. 



The inventive synchronization of elementary streams being 
located out of the main AV multiplex on a optical disc is 
25 described in the following. 

Synchronization is thought of for two different cases. The 
first case is the synchronization of components concerning 
their relative relation in time. It is determining to which 

30 time the separate components start and end. The second case 
is the synchronization of components concerning their 
switching. It is determining points in time and in the bi- 
nary stream were the decoding of one component can be sub- 
stituted by decoding another component. The switching of a 

35 video components for example is useful for multi-angle ap- 
plications . 
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The preferable medium for such an application will be beyond 
DVD capabilities e.g. a blu-ray disc to provide the transfer 
rate needed for out of multiplex bit-stream reading. 

5 The out of multiplex semantic provides the ability to play- 
back blu-ray disc AV streams combining different elementary 
streams inclusive a combination of streams coming from disc 
and via internet or another storage medium. E.g. there 
could be a playback of a movie where the video stream is 
10 read from a blu-ray disc the audio stream is read from a 
hard disc and the sub-title is read from an internet ser- 
vice . 

The description for out of multiplex AV material consists of 
15 different files including navigation files and stream files. 
The navigation files provide all entry points for navigation 
within the stream files as there are points in the time 
axis, angle of view, sub-titles and audio channels etc. In 
a higher layer of the navigation files there exists a list 
20 of play items explaining all stream parts belonging to a 
playback of the AV material . 

In the following a generic syntax for higher level naviga- 
tion files is described. In the tables 1 till 4 only those 
25 elements are outlined which are mandatory for this inven- 
tion, i.e. additional elements not described here as well as 
different orders may be possible. 

As defined within Table 1 a playback element on disc which 
3 0 might be any part of a movie (e.g. a chapter) is described 

by means of a List__of .JPlayltems . It consist of length indi- 
cating the complete length of the structure in byte, play- 
Items indicating the number of PlayltemO elements and sub- 
Playltems indicating the number of SubPlayltems ( ) within the 
35 List_of_PlayItems structure. While at least one Playltem is 
mandatory for a List_of_ Playltems the SubPlayltems are op- 
tional. The time axis of SubPlayltem(s) is referring to the 



WO 2004/034398 PCT/EP2003/010801 

5 

* time axis of the Playlteiti(s) - 



Table 1 : List_of_Playltems - Syntax 



Syntax 


Mn of hitQ 


IWlnfimonif* 

lull IC1IIUI 1 lw 


List of PlayltemsQ { 






length 


1 O 


i limQhf 
Ull 1 lOUl 


playltems 


1 D 


Ull lloUl 








subPlayltems 


16 


uimsbf 








for (n =1 ; n < playltems; n++) { 






PlayltemO { 


















for (n =1 ; n< subPlayltems; n++) { 






SubPlayltemO { 












> 












} 







5 As defined within Table 2 a Playltem structure consists of 
length indicating the complete length of the structure in 
byte, StreamFile describing a link to the elementary stream 
file, Start_time describing the presentation start time of 
the Playltem within the StreamFile and End_time describing 
10 the presentation end time of the Playltem within the Stream- 
File. 

Additionally it consists of the element "reserved" to pre- 
serve byte alignment, Seamless_presentation_f lag indicating 
15 if this Playltem provides seamless transitions between com- 
ponents- The Seamless_presentation_f lag has the same mean- 
ing as within the SubPlayltem (Table 3) and is explained in 
more details there. 
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Table 2: Playltem - Syntax 



Syntax 


NO. OT DltS 


Mnemonic 


p]o\/ltom A J 






lennth 


16 


uimsbf 

UII 1 IOW 1 


StreamFile 












Start_time 


32 


uimsbf 


End_time 


32 


uimsbf 








reserved 


11 


bslbf 


Seamless presentation flag 


1 


bslbf 








> 







As defined within Table 3 a SubPlayltem structure is very 
similar to the structure of a Playltem. It consists of 
5 length indicating the complete length of the structure in 
byte, StreamFile describing a link to the elementary stream 
file, Start_ time describing the presentation start time of 
the Playltem within the StreamFile and End_time describing 
the presentation end time of the Playltem within the Stream- 
10 File. 

Additionally it consists of the element "reserved" to pre- 
serve byte alignment, Seamless_presentation_f lag indicating 
if this Playltem provides seamless transitions between com- 
15 ponents, Stream_path_end indicating the end of a sub stream 
path and SubStream_type an one 8-bit field indicating the 
type of sub stream path given for the SubPlayltem. 
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Table 3: SubPIayltem - Syntax 



Syntax 


No. of bits 


Mnemonic 


SubPlayltemO { 






length 


16 


uimsbf 


Stream File 












Start__time... 


32 


uimsbf 


End_time... 


32 


uimsbf 








reserved 


6 


bslbf 


Seamless_presentation_flag 


1 


bslbf 


Stream_path_end 


1 


bslbf 


SubStream__type 


8 


bslbf 








} 







As defined within Table 4 the SubStream_type defines the 
type of SubPIayltem. This can be an auxiliary audio stream 
path for audio dubbing, a video stream path, an audio stream 
5 path, a subtitle stream path or a graphics stream path. The 
value 6 till 255 can be used for future stream path formats. 



Table 4: SubStream_type 



SubStream„type 


Meaning 


0 


reserved for future use 


1 


Auxiliary audio stream path 


2 


Video stream path 


3 


Audio stream path 


4 


Subtitle stream path 


5 


Graphics stream path 


6-255 


reserved for future use 



10 An alternative definition of the SubStream_type is possible 
as shown in Table 5. The advantage of divining an auxiliary 
transport stream path instead of different elementary stream 
paths is, that any elementary stream can again be embedded 
within another multiplex stream, i.e. several different sub- 

15 titles may be multiplexed together within a single file. 
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Table 5: SubStreamJype 



SubStream_type 


Meaning 


0 


reserved for future use 


1 


Auxiliary audio stream path 


2 


Auxiliary transport stream path 


3-255 


reserved for future use 



Seamless_presentation_j£ lag supports the case of synchroniza- 
tion concerning seamless switching of components during 
5 playback. 

The Seamless_presentation__f lag is located within the Play- 
Item/ SubPlay I tern and indicates if the transport stream ref- 
erenced to by Playl tern/ SubPlayl tern is supporting seamless 
10 transitions between its elementary streams. If Seam- 

less_presentation_f lag is set "true", all elementary streams 
. within the transport stream obey the seamless transition re- 
strictions. 

15 The seamless transition restrictions are providing Splice 
Points within the elementary stream. Splice Points are 
points at which the decoding of one elementary stream can be 
stopped and the decoding of another can be started without 
having any noticeable effect during playback, presuming that 

2 0 both elementary streams have been encoded by obeying the 

seamless transition restrictions . The generation of Splice 
Points is reached during encoding by defining a mandatory 
GOP raster, e.g. a fixed GOP length, and by limiting the re- 
quired buffer size, to avoid buffer overflows when switching 

25 different elementary streams during decoding. 

A typical application for seamless transitions are multi- 
angle videos. A multi-angle video provides different camera 
angles for a video e.g. one camera within the race-car, an- 
30 other showing the box, one showing the finish line etc. 
When the multi-angle video is played back the user can 
switch seamless between this different camera tracks . 
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Stream__path_end and SubStream__type support the case of syn- 
chronization concerning the relative relation of components 
in time. 

5 

Stream_path_end: This flag indicates the end of a stream 
path. A stream path is an ordered set of SubPlayl terns . It 
defines the number and order of all SubPlayltems belonging 
to the stream path. The first stream path starts with the 

10 first SubPlayltem and ends with the first SubPlayltem having 
the Streaitupath_end flag set "true" . The second stream path 
starts with the first SubPlayltem following the previous 
stream path and ends with the first SubPlayltem having the 
Stream_path_end flag set "true", and so on. The last stream 

15 path ends with the last SubPlayltem. Setting the 

StreanLj>ath_end flag "true" for the last SubPlayltem of the 
PlayList is optional. All SubPlayltems of a stream path 
shall have the same SubPlayItem__type . 

20 Playing back a stream path will decode the consecutive set 
of SubPlayltems belonging to the stream path including ei- 
ther the first SubPlayltem having the Stream_path_end flag 
set "true" or the last SubPlayltem of the PlayList. 

25 The Transport Stream (TS) is organized according to the 

MPEG-2 Systems standard as specified in ISO/IEC 13 818-1, Ge- 
neric coding of moving pictures and associated audio infor- 
mation. However, in order to guarantee a conflict free play- 
back during decoding and at the same time a late and inde- 

30 pendent addition of content, e.g. the adding of auxiliary 
subtitles after the main authoring has already been final- 
ized, the following rules of restrictions for the transport 
streams are necessary: 

35 The TS of Playltems must contain a Program Association Table 
(PAT) and Program Map Table (PMT) . They provide the PID ta- 
ble and language information for every TS component (elemen- 
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tary stream) within the multiplex. 

The TS of SubPlayltems contains auxiliary streams. The pre- 
ferred format of auxiliary streams again is a TS. 
Within the TS of a SubPlayltem no PAT is allowed. This pre- 
5 vents conflicts with the TS of the Playltem. 

The TS of a SubPlayltem must contain a PMT whose PID is not 
used within the TS of Playltems . 

All PID associated to elementary streams within all TS for 
Playltems as well as SubPlayltems must be different. 

10 

The advantage of the TS rules listed above is, that a single 
PID filter is sufficient for decoding all elementary 
streams. When violating the TS rules a PID re-stamping be- 
fore buffering the input data or the separating of different 
15 buffers becomes necessary. 

The invention has several advantages: 

It provides more flexibility for the authoring of pre- 
recorded blue-ray discs. E.g. after the f inalization of the 

20 main AV multiplex, additional languages can easily be added 
to the disc at a later time. This is because an auxiliary AV 
multiplexes containing e.g. audio and sub- title information 
in other languages can be prepared independently and added 
to the disc image without changing the basic AV multiplex 

25 (no recoding, re-multiplexing required) . 

There are no limitation for the maximum number of audio, 
subtitle, video angles, AV formats etc. caused by the band- 
width given for the multiplex. The limitation is simply de- 
3 0 fined by the volume capacity. 

It allows optional binding of external sources to disc con- 
tent e.g. a special sound track from hard disc or internet 
server, resulting in added value by binding new streams to 
35 pre-recorded discs . 

Furthermore, rules are provided for simplified decoding of 
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elementary streams coming from different transport streams 
and for seamless stream switching (e.g. for multi-angle vid- 
eos) across different transport streams. 



